Mnist Data with noising

$256$ discrete values between $0$ and $255$, every value is represented.

Evaluation comments

For the metrics the ouput $y \in [-1,1]$ was not trasformed nor rounded.


absolute $\epsilon$-accuracy


diff <- |target_set - predicted_set|        // pixelwise difference stored as (10000, 784)
accu <- 0                                   // accumulator
loop elem in  diff                          // for each element in diff i.e. for each number, image (784,)
    accu <- |{i ∈ elem : elem < ε }| / 784  //count how many elements are > ε and average over pixels, i.e. divide by 784

accu <- accu/10000                          // Average over examples in image set

absolute $\epsilon$-outliers


diff <- |target_set - predicted_set|        // pixelwise difference stored as (10000, 784)
accu <- 0                                   // accumulator
loop elem in  diff                          // for each element in diff i.e. for each number, image (784,)
    accu <- |{i ∈ elem : elem > ε }| / 784  //count how many elements are > ε and average over pixels, i.e. divide by 784

accu <- accu/10000                          // Average over examples in image set

squared $\epsilon$-accuracy


diff <- (target_set - predicted_set)^2       // pixelwise difference stored as (10000, 784)
accu <- 0                                   // accumulator
loop elem in  diff                          // for each element in diff i.e. for each number, image (784,)
    accu <- |{i ∈ elem : elem < ε }| / 784  //count how many elements are > ε and average over pixels, i.e. divide by 784

accu <- accu/10000                          // Average over examples in image set

squared $\epsilon$-outliers


diff <- (target_set - predicted_set)^2       // pixelwise difference stored as (10000, 784)
accu <- 0                                   // accumulator
loop elem in  diff                          // for each element in diff i.e. for each number, image (784,)
    accu <- |{i ∈ elem : elem > ε }| / 784  //count how many elements are > ε and average over pixels, i.e. divide by 784

accu <- accu/10000                          // Average over examples in image set

$\epsilon$-accuracies for $\epsilon= 1/256$, over noise and $\epsilon$-outliers for $\epsilon= 0.5$,

$\epsilon$-accuracies for $\epsilon= 1/256$, over noise and $\epsilon$-outliers for $\epsilon= 1$,

$\epsilon$-accuracies for $\epsilon= 1/100$, over noise and $\epsilon$-outliers for $\epsilon= 1$,

$\epsilon$-accuracies for $\epsilon= 1/10$, over noise and $\epsilon$-outliers for $\epsilon= 1.5$,

mse: mean squared error

$\operatorname{mse} =\frac{1}{m} \sum_{j=1}^{m} \frac{1}{n} (Y_j-\hat{Y_j})^2 = \frac{1}{m} \sum_{j=1}^{m} \frac{1}{n} \sum_{i=1}^{n} (y_i-\hat{y_i})^2$
where:

  • $m$ number of examples i.e. $10000$
  • $\mathbb{Y}=\{Y_1, Y_2, \cdots, Y_m\}$ target image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $\mathbb{\hat{Y}}=\{\hat{Y}_1, \hat{Y}_2, \cdots, \hat{Y}_m\}$ predicted image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $n$ number of pixels i.e. $784$
  • $Y_j=\{y_{j 1},y_{j 2}, \cdots, y_{j n}\}$ is the target data of image $j$ i.e. $(784,1 )$ vector
  • $\hat{Y_j}=\{\hat{y}_{j 1},\hat{y}_{j 2}, \cdots, \hat{y}_{j n}\}$ is the prediction of image $j$ i.e. $(784, 1)$ vector

mae: mean absolute error

$\operatorname{mae} =\frac{1}{m} \sum_{j=1}^{m} \frac{1}{n} |Y_j-\hat{Y_j}| = \frac{1}{m} \sum_{j=1}^{m} \frac{1}{n} \sum_{i=1}^{n} |y_i-\hat{y_i}|$
where:

  • $m$ number of examples i.e. $10000$
  • $\mathbb{Y}=\{Y_1, Y_2, \cdots, Y_m\}$ target image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $\mathbb{\hat{Y}}=\{\hat{Y}_1, \hat{Y}_2, \cdots, \hat{Y}_m\}$ predicted image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $n$ number of pixels i.e. $784$
  • $Y_j=\{y_{j 1},y_{j 2}, \cdots, y_{j n}\}$ is the target data of image $j$ i.e. $(784,1 )$ vector
  • $\hat{Y_j}=\{\hat{y}_{j 1},\hat{y}_{j 2}, \cdots, \hat{y}_{j n}\}$ is the prediction of image $j$ i.e. $(784, 1)$ vector

Average maximal difference

Average over: the maximal absolute diference between pixels in each example. i.e. $\underset{i}{mean}(max(|\mathbb{Y}_i-\mathbb{\hat{Y}}_i|))$

Maximal difference

Maximum over: the maximal absolute diference between pixels in each example. i.e. $max(max(|\mathbb{Y}_i-\mathbb{\hat{Y}}_i|))$

where:

  • $m$ number of examples i.e. $10000$
  • $\mathbb{Y}=\{Y_1, Y_2, \cdots, Y_m\}$ target image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $\mathbb{\hat{Y}}=\{\hat{Y}_1, \hat{Y}_2, \cdots, \hat{Y}_m\}$ predicted image set, i.e. set of $m$ images each image is represented by a vector of length $n$
  • $n$ number of pixels i.e. $784$
  • $Y_j=\{y_{j 1},y_{j 2}, \cdots, y_{j n}\}$ is the target data of image $j$ i.e. $(784,1 )$ vector
  • $\hat{Y_j}=\{\hat{y}_{j 1},\hat{y}_{j 2}, \cdots, \hat{y}_{j n}\}$ is the prediction of image $j$ i.e. $(784, 1)$ vector

Error distribution


absolute $\epsilon$-error


diff <- |target_set - predicted_set|        // pixelwise difference stored as (10000, 784)
diff <- flatten dif                         // becomes vector of 10000 * 784 different error values
diff <- filter all elem < ε                 // all elements smallet than ε are taken out of the list
create histogram(diff, bin_size)

squared $\epsilon$-error


diff <- (target_set - predicted_set)^2        // pixelwise difference stored as (10000, 784)
diff <- flatten dif                         // becomes vector of 10000 * 784 different error values
diff <- filter all elem < ε                 // all elements smallet than ε are taken out of the list
create histogram(diff, bin_size)

Histogram of bin size $32$ for $\epsilon= 1/256$, over noise

Histogram of bin size $32$ for $\epsilon= 0.039$, over noise

Histogram of bin size $32$ for $\epsilon= 0.5$, over noise

Histogram of bin size $32$ for $\epsilon= 1$, over noise

[NbConvertApp] Converting notebook New_comparissons.ipynb to html
[NbConvertApp] Writing 9970453 bytes to ./No_code/New_comparissons.html